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Abstract 

Cameras with rolling shutters are becoming more com¬ 
mon as low-power, low-cost CMOS sensors are being 
used more frequently in cameras. The rolling shutter 
means that not all scanlines are exposed over the same 
time interval. The effects of a rolling shutter are notice¬ 
able when either the camera or objects in the scene are 
moving and can lead to systematic biases in projection 
estimation. We develop a general projection equation 
for a rolling shutter camera and show how it is affected 
by different types of camera motion. In the case of 
fronto-parallel motion, we show how that camera can 
be modeled as an X-slit camera. We also develop ap¬ 
proximate projection equations for a non-zero angular 
velocity about the optical axis and approximate the 
projection equation for a constant velocity screw mo¬ 
tion. We demonstrate how the rolling shutter effects 
the projective geometry of the camera and in turn the 
structure-from-motion. 

1 Introduction 

The motivation for this paper is the use of inexpen¬ 
sive camera sensors for robotics. Maintaining accu¬ 
racy in this low-cost realm is difficult to do, especially 
when sacrifices in sensor design are made in order to 
reduce cost or power consumption. For example, as 
of 2004, many of the CMOS chips manufactured by 
OEMs and integrated into off-the-shelf cameras do not 
have a global shutter.^ Unlike CCD chips with interline 
transfer, the individual pixels in a CMOS camera chip 
typically cannot hold and store intensities, forcing a so- 
called rolling shutter whereby each scanline is exposed, 
read out, and in the case of most Firewire cameras, is 
immediately transmitted to the host computer. Fur¬ 
thermore, for cameras connected via a Firewire bus, 
the digital camera specification P requires that cam¬ 
eras distribute data over a constant number of packets 
regardless of the framerate. Thus, if the camera has no 
on-board buffer, the overall exposure for a single frame 
is inversely proportional to the framerate. 

^Fortunately, some progress has been made in CMOS chips 
with a global shutter, e.g. see 1^ . 


However, most structure-from-motion algorithms 
(see e.g. 0lsi) make the assumption that the exposure 
of frames is instantaneous, and that every scanline is 
exposed at the same time. If either the camera is mov¬ 
ing or the scene is not static then the rolling shutter 
will induce geometric distortions when compared with 
a camera equipped with a global shutter. Consider, for 
example. Figure ^ which shows the distortion present 
in the image of a fronto-parallel checkerboard when a 
rolling-shutter equipped camera is rotating about the 
optical axis. 

The fact that every scanline is exposed at a differ¬ 
ent time will lead to systematic biases in motion and 
structure estimates.Thus, in considering the design of 
a robotic platform,which often by necessity is in mo¬ 
tion (for example a helicopter) one needs to balance 
accuracy, cost and power consumption. The goal of 
this paper is to obviate this trade-off. By modeling 
the rolling-shutter distortion we mitigate the reduc¬ 
tion in accuracy and present a framework for analyz¬ 
ing structure-from-motion problems in rolling-shutter 
cameras 

We define a general rolling-shutter eonstraint on im¬ 
age points which is shown to be true in the case of 
an arbitrarily moving rolling-shutter camera. Using 
this constraint we derive a projection equation for the 
case of constant velocity fronto-parallel motion, which 
is exact when the angular velocity about the optical 
axis is zero and furthermore equivalent to a so-called 
“crossed-slits” camera. The projection equation is ex¬ 
pressed in terms of the perspective projection plus a 
correction term, and we show that bounds on its mag¬ 
nitude yield safe regions in the space of depth vs. ve¬ 
locity where it is sufficient to use pin-hole projection 
models. For domains where the pin-hole model is insuf¬ 
ficient, we demonstrate a simple method to calibrate a 
rolling shutter camera. With new projection equations 
we re-derive the optical flow equation, and we present 
experiments in simulation data. 

Despite the prevalence of rolling-shutter cameras 
for consumer electronics and web cameras, little has 
been done to address issues related to structure-from- 
motion. One of the few examples which specifically 
model the rolling-shutter is the recent work by Levoy 
et al. m who have constructed an array of CMOS- 
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Figure 1: These four images show a physically recti¬ 
linear checkerboard as seen from a rolling shutter. The 
plane is 0.5 meter from the camera, the camera has a 
40 degree field of view, is operating at 30 frames per 
second. From left to right, top to bottom, the camera 
is shown rotating about the optical axis at |^, | and 
1 revolution per second, respectively. 


equipped cameras for high speed videography. The au¬ 
thors propose to construct high framerate video by ap¬ 
propriate selection of scanlines from an array of cam¬ 
eras, while compensating for the rolling shutter effect. 
We are not aware of any research to specifically ad¬ 
dress the rolling shutter phenomenon for structure- 
from-motion. However, such cameras are modeled in 
the study of non-central cameras (Hj, that is, those cam¬ 
eras which do not have a single effective viewpoint, or 
focus. In fact, they are closely related to a specific type 
of camera variously known as X-, crossed-, or two-slit 
cameras, discussed in m, m and [7|, and generalized 
in El. Instead of a single focus, all imaged rays coin¬ 
cide with two slits. We show here that under certain 
circumstances, the rolling-shutter camera is a two-slit 
camera. However, the difference in focus between this 
paper and most of the work on two-slit cameras is that 
two-slit cameras have heretofore been simulated by se¬ 
lection of rows or columns from a translating camera. 
Therefore, calibration and motion estimation can be 
done by traditional means using all the information in 
each individual image, whereas we do not have that 
luxury. 

In related works, Pless (Hj first described the epipo- 
lar constraint and infinitessimal motion equations for 
arbitrary non-central cameras, where cameras are rep¬ 
resented by a subset of the space of lines. Pajdla [3 
has shown that the pushbroom camera pj, which is 


approximates some satellite and aerial imagers, is just 
one type of two-slit camera. Latzel and Tsotsos |3 
have studied motion detection in the case of a related 
sensor problem, namely interlaced cameras. 


2 Model of Rolling Shutter 
Cameras 

In this section we describe an equation for modeling 
cameras with rolling-shutters. When camera motion 
in the direction parallel to the fronto-parallel plane is 
taking place we show how this effects the projection 
and develop a projection equation independent of time. 
An interpretation of progressive scan as a so-called “X- 
slit” camera (meaning that instead of obeying a pin¬ 
hole projection model the camera is modelled by two 
slits) results from certain types of motion. 

We begin by describing the operation of the MDCS 
Firewire camera, a specific rolling-shutter camera sold 
by Videre Design. The principles described here ap¬ 
ply to other cameras with rolling shutters, with only 
minor differences in some of the controllable variables. 
The Firewire bus has a clock operating at 8Khz so the 
camera can send no more than 8000 packets per second. 
The HDC specification for digital cameras Q] mandates 
that frames sent over the bus should be equally spread 
out over these 8000 packets. The MDCS camera does 
not have an on-board buffer of sufficient size to store a 
whole frame, and therefore it must send data as soon as 
it is read from scanlines. The scanlines are sequentially 
exposed, read in, and immediately sent over the bus 
such that the total time of exposure of one frame is in¬ 
versely proportional to the framerate. The framerate(/ 
frames/sec), the exposure length of one scanline (e /is), 
the rate at which scanlines are exposed (r rows/yus), 
and any delay between frames (dyus)are the variables 
which control exposure of the scanlines^. All of the 
e, r, and d will in general be dependent in some way 
on /, the framerate, and the camera. An example of 
how a set of scanlines is exposed is shown in Figure [2l 
We assume that the exposure time within a scanline 
is instantaneous, i.e. that the peaks in Figure (21 have 
zero width but integrate to some non-zero constant. 
The effect of e being non-zero is motion blur within 
the scanline, but has no geometric effects. 

In a rolling-shutter camera each row of pixels is 
scanned in sequential order over time to create an im¬ 
age. We can formalize a camera model by noting that 
each row corresponds to a different instant in time, as- 

^The delay d would normally be 0 for IIDC cameras with a 
rolling-shutter and without an on-board frame buffer; for general 
cameras, though, this ought to be verified. 
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Figure 2: Rolling-shutter cameras expose scanline se¬ 
quentially; for some Firewire cameras the total expo¬ 
sure and the scanning rate may be inversely propor¬ 
tional to the framerate. The scanline currently being 
scanned depends on the function Vcam{t)^ defined in 
o, which we assume is linear in t. 

suming that the exposure is instantaneous. We suppose 
that for a frame whose exposure starts at to, the row 
index as a function of time is described by: 

Vcam{to + t) = rt-Vo, (1) 

where r is the rate in rows per microsecond, and vq 
is the index of the first exposed scanline. The sign 
of r depends on whether scanning is top-to-bottom or 
bottom-to-top in the sensor. For now let us assume 
that to = 0. 

For an ideal perspective camera, the projection of 
a point as a function of time is determined using the 
perspective camera equation: 

q{t) = n{Pit)X) (2) 

where X = (x, z, 1) G represents, in homogeneous 
coordinates, some static point in space; P(t) is a cam¬ 
era matrix as a function of time: 

P{t) = K [R{t) T{t)] and TT{x,y,z,l) = (f, |). 

Here, R{t) is an element of SO(3), the space of orien¬ 
tation preserving rotations; T(t) G so that V{t) = 
—R(t)T(t) is the camera’s viewpoint at time t in a 
world coordinate system; and finally K is an upper- 
triangular calibration matrix. This configuration is 
shown in Figure 01 

Given that we have imaged some point X at say 
(i4,n), it must be the case that: 

7ry{P{tc)X) = rtc-vo = V (3) 



Figure 3: Left: the projection of a point in a moving 
perspective camera generates some curve q{t) in the 
image; Right: in the reference frame of the camera, 
the point is apparently moving and the is capture by 
the scanline at some tc- 

general rolling-shutter constraint. The left hand side 
describes the curve of the projected point over time, 
while the right hand side of the equation represent the 
scanline as a function of time. The scanline captures 
the curve of the projected point at some time instance 
tc. 

If the camera is stationary, i.e. P{t) is independent 
of t, then the left hand side of equation 0 is inde¬ 
pendent of time so the image of X is independent of 
tc and the resulting projection is the usual perspective 
one. However, in general the camera will be moving 
and the resulting projection will depend on tc in some 
way. Under certain assumptions on P(t) we can solve 
analytically for G, substitute it into the left hand side 
of 0 and obtain a projection formula. The goal of the 
next section is to derive specific projection formulas in 
common situations. 

2.1 Fronto-parallel motion 

Suppose that a calibrated camera undergoes a constant 
angular velocity uj and a linear velocity v. A linear 
approximation of P{tc) about tc = 0 is given by: 

P{tc) ~ LC [ (/+ tc <^)R(0) T(0)+ntc]. 

When cj = 0, i.e., when there is no angular velocity, 
the linearization is exact; otherwise it is an approxi¬ 
mation. Let us analyze the case of constant velocity 
fronto-parallel motion; that is, 

V = [vx,Vy,0]^ and lj = [0,0, 


for some time tc, and where iVy represents the y- 
component of the projection tt. Equation (H is the 
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Substituting the approximation of 0 into equation 
the rolling-shutter constraint, yields an equation 
linear in tc- Assuming without loss of generality that 











R{0) = /, T(0) = [0, 0, 0]^, and K = I, then after sub¬ 
stituting the solution to tc back into equation q{tc), we 
obtain: 

~x I y+VQZ ivx-Wzy) ~ 

_ 2: ' {rz — Vy—wzx) z / a\ 

~ y , y+VQZ (vy+Wzx) V ) 

_z ' {r z—Vy—w zx) z 

—1“ ^correction* 

This is the image of the point {x^y^z) in the frame of a 
rolling-shutter eamera starting at t = 0. If cj = 0, this 
equation is exact. Therefore, we arrive at a projection 
equation for a moving camera with a rolling-shutter 
which is independent of time. We write the projection 
equation so that it is more obvious what the effects of 
the rolling-shutter are, namely that the resulting pro¬ 
jection equals the ideal perspective projection plus a 
correction term which is proportional to the optical 
flow ^ vy+wzx ^T ^ perspective camera. Note 

that as the rate of scan, r, goes to oo, the limit of 
the correction term becomes [ 0 , 0 ]^, corresponding to 
a camera with a global instantaneous shutter. 

Thus, what a camera with a rolling-shutter becomes 
is a camera whose projection geometry is parameter¬ 
ized by its velocity. When stationary, the camera obeys 
the pinhole projection model. However, when under¬ 
going linear motion with constant velocity, one can 
show that the camera becomes a crossed-slits camera. 
For any point {u^v) we may invert 0) up to an un¬ 
known ^ and demonstrate that all such inverse im¬ 
age coincide with two lines: the first being the line 
through ( 0 , 0 , 0 ) and {vx, 0 ); and the second through 
the points (± 1 ,^^^,^). As Vx and Vy approach 0 , 
these two slits coincide at (0,0,0). The case when 
ooz = Vx = 0 is shown in the left part of Figure ^ 
in the middle we show the case when Vx and Vy are ar¬ 
bitrary, and Uz = 0] finally in the right part of Figure 
0 we show that the resulting geometry when 7 ^ 0 , in 
which case the camera no longer obeys the crossed-slits 
camera model. 

2.2 Domain of applicability 

Under what condition is the perspective model a suf¬ 
ficient modei for roiiing-shutter cameras? If error in 
feature iocation is on the order of one pixei, then we 
wouid expect distortions due to the roiiing-shutter to 
be subsumed in noise. Thus, the maximum vaiue that 
the correction term can obtain shouid be iess than one 
pixei iength in order for a perspective camera modei 
to be an adequate representation of the roiiing shut¬ 
ter camera. The correction term varies as a function 
of veiocity and depth 2 :. In order to determine when 
this correction term needs to be factored in, and a per¬ 
spective modei is no ionger suitabie, these parameters 


q rolling - q{tc) 


must be evaiuated in iight of the fixed parameters of 
the camera and the shutter rate r. Given that Sq, is 
the iength of a pixei, it can be seen that: 

— Vy < Z (5) 

Sc^r 

must be satisfied in order for the roiiing shutter effects 
to be inconsequentiai. However, when this inequaiity 
does not hoid, then the roiiing shutter projection modei 
shouid be used or there will be a bias in estimations. 
This creates a limit line, as a function of Vy and z, 
based on the shutter rate and s^^ as shown in Figur^S 
When the values are such that the camera is operating 
well above this line, a rolling shutter model should be 
used. 



Figure 5: The limit line for a 640 x 480 image for 
different framerates. Blue represents 3.75fps, green is 
7.5fps, and red is 15fps. 


2.3 General rotation and translation 

Generalizing the motion for any T = [vxVyVz]'^ and an 
arbitrary rotation matrix R we can see the relation of 
the rolling-shutter camera projection to G for different. 
We discussed the linearity of the fronto-parallel case 
previously and can be see in ©• When there is a non¬ 
zero velocity along the principle axis of the camera, 
Vz, the relation now becomes quadratic. Using a root 
solver, we can find a solution of tc to make a rational 
projection equation. Different types of motion and the 
relation of the projection function to tc follow from the 
rolling-shutter rate o and (|3|). This relation and its 
effects on the projection are shown in Figure 0 

The linearized P{t) and the resulting projection 
equation for rolling shutter camera can be used given 
any type of camera motion. In certain cases tc is a 
solution to a linear equation and in other cases, tc is 
a solution to a quadratic. In both instances tc can be 
found either directly or using a root solver and then 
used in the general projection equation to find a pro- 
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Figure 4: Left: When only Vy ^ 0, the camera becomes an orthogonal crossed-slit camera; middle: when Vx^Vy 
are non-zero, the camera is a general crossed-slit camera; and right: when cj 7 ^ 0 , the approximating camera is no 
longer a crossed-slit camera. 


Case 

Solution to tc 

Projection equation 

Constant velocities v = [vx^Vy^O]^ 
w = [0^0, Wz], and linear 
approximation of P{t). 

Results in a linear 
equation which can be 
solved for tc- 

Equation Q). 

Constant velocities v = [ 0 , 0 ,^;^], 
w = [ 0 , 0 , 0 ], and linear 
approximation of P{t). 

Results in a quadratic 
equation for tc, choose 
smallest positive root. 

The smallest positive root of tc when 
substituted into q{tc) yields an analytic 
equation for the projection without tc- 

Constant velocities v = [vx^Vy^Vz], 
w = [wx^Wy^Wz], and linear 
approximation of P{t). 

Also results in a 
quadratic equation in tc- 

Using the solution for tc in the general 
projection g(tc) leads to a projection 
model is analytic and independent of tc- 

Constant velocities v = [vx^Vy^Vz]^ 
w = [wx^Wy^Wz], without linear 
approximation of P{t). 

Nonlinear equation for 

tc; can be solved with a 
root finder (when 
(x, y, z) are known). 

The projection equation q{tc) is nonlinear 
in tc- 


Table 1: Different assumptions on the motion, and our choice in whether to linearize P(t) lead to different types 
of solutions for tc and the projection q{tc). 


jection model of the rolling shutter that is indepen¬ 
dent of tc- The resulting projection model will only be 
dependent on the velocity of the camera, the internal 
parameters, and the shutter rate. 


3 Calibration 

In this section we discuss the calibration of a Firewire 
camera with rolling shutter, namely the MDCS cam¬ 
era from Videre design. Our goal is to determine the 
constant r, the rate of scan, and determine if d, the de¬ 
lay between frames, is zero. If we find that r = '^r//, 
where Ur is the number of rows and / is the framerate, 
then we can conclude that d = 0 . 

Since the camera exposes only several neighboring 
scanlines at a time, we can indirectly measure r by 


placing an LED flashing at known frequency in front 
of the camera and measuring the peak frequency of ver¬ 
tical slices of the image in the frequency domain. In a 
simple experiment we placed the camera in front of a 
green LED, removing its lens to ensure that all pixels 
are exposed by the LED, and draped a dark cloth over 
the setup so as to eliminate ambient light. An exam¬ 
ple of a captured image is shown in Eigure|3| we were 
unable to achieve even distribution of exposure over 
the length of the scanline. Eor any choice of framerate 
and frequency, we selected and summed a subset of the 
columns from each image, yielding for any one time 
ti the vector thereby constructing an image 

I{y, t) for each experiment whose independent variable 
is the pair of framerate and frequency. The duration 
of exposure, which is a configurable parameter, affects 
the width of the horizontal bars in Eigure 13 though 
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3.75 FPS 


it does not affect their period. To estimate the fre¬ 
quency we take the two-dimensional Fourier transform 
of /(^, t), obtaining I{v^uo) which we marginalize over 
uo to yield The estimate of r is the location of 

the highest frequency peak (after high frequency com¬ 
ponents are ignored because the image is not a sine 
wave). Thus, using the image based frequency of the 
light flashes in conjunction with the actual frequency of 
the LED flashes from the pulse generator, we were able 
to approximate the rate of the rolling shutter for the 
camera. It can be seen in Figure |3| that it is sufficient 
to model our camera as having no delay 


FPS 

calibrated time 
(sec/row) 

ideal time 
(sec/row) 

3.75 

0.00110 ± 0.00050 

0.00110 

7.5 

0.00063 ± 0.00050 

0.00056 

15 

0.00029 ± 0.00050 

0.00028 


Table 2: The calibrated time to scan a row in relation 
to the ideal time to scan a row given the framerate. 
Note that the accuracy of the pulse generator frequency 
is taking into account in the offset number 



50 100 150 200 250 300 50 100 150 200 250 300 



Figure 7: The positive frequencies of the marginalized 
2D Fourier Transform of the LED capture image for 
framerate of 3.75 FPS. The blue shows the results using 
a pulse generator frequency (LED frequency) of 103Hz. 
The red shows the results for a pulse generator fre¬ 
quency of 20Hz. The green shows the results for pulse 
generator frequency of 11 Hz. 


at to = 0. For the case of fronto-parallel motion, i.e. 
V = (vx^Vy^O)^ and uj = (0,0, Cc;;^), and linearization of 
to, this procedure yields the following optical flow: 

rz friip -h uOzVVp 

vVxWz + rz{r — Vp) \rvp + ujzviip 

where {up^Vp) is the optical flow for perspective cam¬ 
eras in the case of fronto-parallel motion: 


qiu,v) = 


Figure 6 : Left: an example image taken during calibra¬ 
tion; right: an example of the spatio-temporal image 
I{y,t) used to estimate the rate r. 


{ilpfVp) — [ ^ ^ - 1 -^ 2 :'^] • 

In the case of general motion and linearization of P(t), 
the resulting equation for optical flow remains analytic 
but is significantly more complicated. 


3.1 Optical flow equations 

To determine the optical flow equations for rolling- 
shutter cameras, we make the assumption that the pa¬ 
rameterized camera can be sampled infinitely often in 
time. Furthermore, for ease of presentation we will 
assume from now on that = 0 ; in general this may 
induce some tc < 0. To determine the flow we solve the 
rolling-shutter constraint using the camera P(to + tc), 
and having found and substituted tc as well as expres¬ 
sions for X and y in terms of the image coordinates 
and z, we differentiate with respect to to and evaluate 

^Frequency of they LED frequency (pulse generator fre¬ 
quency) was varied from 2.5Hz to 103Hz. Not all framerates 
tested the complete span of LED frequencies 


4 Simulation 

In section o we determined those conditions under 
which it may be wise to take into account distortions 
caused by a rolling shutter. In this section we test this 
claim in simulations under varying speeds and noise 
conditions. We randomly generate a set of 100 points 
and project these points into two cameras according 
to the rolling-shutter camera model for certain Vy ^ 
0, all others zero. The cameras are placed randomly, 
but on the order of 10 meters from the point cloud. 
Then, assuming a camera sampling at 15 frames per 
second, with scanning rate inversely proportional to 
the framerate, we let Vy = 1.875, 3.75, 5.625 and 7.5 
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Velocity: 1.875 km/h. 



Simulated error (pixels) 


Velocity: 3.75 km/h. 



Simulated error (pixels) 


Velocity: 5.625 km/h. 



Simulated error (pixels) 


Velocity: 7.5 km/h. 



Simulated error (pixels) 
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Figure 8: Plot of reprojection errors, errors in rotation and translation, and for varying velocities. 


km/h. For each velocity we perform experiments with 
pixel noises a = 0.5, 1.33, 2.16, 3, 3.83 and 4.66 pixels. 

For each instantiation of the experiment, we perform 
bundle adjustment on the image points using a pro¬ 
jection model incorporating the rolling shutter (blue, 
solid lines) and the perspective projection model (red, 
dashed lines). The errors, plotted in Figure|Hl are: left: 
reprojection error, middle: the error of the estimate R 
of rotation when compared with the true Rq , measured 
as II logi?^^^||; and, right: the error of the estimate t 
of translation when compared with the true to, mea¬ 
sured as cos“^ t^to/(||t|| ||to||). 

For high velocities or low noise we find that by com¬ 
pensating for the rolling shutter, one is able to do bet¬ 
ter than if one were to ignore the effect. However, for 
lower velocities and higher noise, the noise effectively 
becomes louder, comparatively, than the deviation due 


to the rolling-shutter. 

5 Conclusion 

Despite the prevalence of rolling-shutter cameras little 
has been done to address the issues related to structure- 
from-motion. Since these cameras are becoming more 
common, the effects due to the shutter rate must be 
taken into account in vision systems. We have pre¬ 
sented an analysis of the projective geometry of the 
rolling-shutter camera. In turn, we have derived the 
projection equation for such a camera and demon¬ 
strated that under certain conditions on the motion 
and internal parameters, using this flow equation can 
increase accuracy when doing structure-from-motion. 

The effect of the rolling-shutter becomes important 
when either the camera or objects in the scene are 
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moving, as the shutter rate can produce distortion if 
not accounted for. The modeling we have presented 
may play an important role for incorporating rolling- 
shutter cameras into moving systems, such as UAV’s 
and ground robots. Given the improved accuracy in 
structure-from-motion, this can aid in scene recovery 
and navigation of systems operating in dynamic worlds. 
Future work includes testing these models on UAV’s for 
recovery of scene structure. 
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